On learning simple neural concepts: from halfspace intersections to neural decision lists

In this paper, we We a close look at the problem of l&ing simple neural concepts under the uniform diseibution of examples By simple neural concepts we mean concepts that can be represented as simple combinations of perceptrons (halfspaces). One such class of concepts is the class of halfspace intersections. By formalizing the problem of learning halfspace in&ections as a sei-coveringproblem. we are led to mnsider the following subproblem: given a set of nonliniarly separable examples, find the largest linearly separable subset of it We give an approximation algorithm for this NP-hard sub-problem. Simulations, on. both linearly and nonlinearly separable functions, show-thai this approximation algorithm works well uirder the uniform dishibution. outperforming the packet algorithm vsed, by many WnsLNctive neural algorithms. Based on this approximation algorithm, we present a p d y method for learning halfspace inferseclions, We also present extensive numericaI results that strongly suggest Ihat this greedy method leams ha l f spa i n t e d o n s under the uniform dishibution of examples. Finally, we infroduce a new class of simple. yet very rich, neural co"cepts that we call neural decision luis. We show how the greedy method can be generalized to handle this class of concepts. Both greedy methods for halfspak intersections and'neural decision lists were vied on rea-world data with very encouraging nsults. This shows that these concepts are not only important from the theoretical point of view, but also in practice.